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TRENDS IN RESEARCH METHODS 



Making Connections: Using 
Social Network Analysis for 
Program Evaluation 

by Todd Honeycutt 



k3ocial network analysis (SNA) is a methodologi- 
cal approach to measuring and mapping relation- 
ships. It can be used to study whole networks, all 
of the ties within a defined group, or connections 
that individuals have in their personal communi- 
ties. The resulting graph-based structures illustrate 
the composition and effectiveness of networks on a 
variety of levels. Programs that can benefit from a 
social network perspective are common in evalu- 
ation research. This brief introduces the concepts 
behind SNA and illustrates how to use this method 
in process and outcome evaluations. 



Uncovering Patterns in Social Networks 

How do physician/nurse interactions affect patient out- 
comes? Are teachers who have better access to profes- 
sional advice more effective instructors? To what extent 
do parents work together to advocate for their children 
and affect school policy? These examples from the fields 
of health and education illustrate evaluation questions 
that involve social networks. 

Social network analysis (SNA) can help map and measure 
human and organizational relationships, both visually 
and mathematically. SNA views relationships in terms of 
nodes and ties — nodes are the individual actors in net- 
works, and ties are the relationships between the actors. 

As part of a process evaluation, SNA can shed light into 
the “black box” of how and why people and groups inter- 
act. The methodology can address questions such as: 

• Which organizations are working together to address 
disparity issues? Which are not? 

• How are financial and other resources distributed 
among organizations promoting consumer advocacy? 



SNA can also be used for outcome evaluation, linking 
characteristics of a network to program outcomes or 
determining whether relationships have changed as a 
program intended. It can address questions like: 

• Are coalitions in which members are more connected 
more likely to have an impact on the long-term care 
system? 

• How does a network evolve so that it is sustainable 
after initial funding ends? 

SNA in Process Evaluation 

SNA opens the black box of a program’s processes, pro- 
viding a qualitative and quantitative assessment of net- 
work relationships. These data can be used to determine 
the relationships among network members, identify 
which members are more “involved,” target interven- 
tions to improve the network, and provide formative 
feedback to program staff. 

Examine relationships. A starting point in using 
SNA involves defining boundaries of the network and 
describing the nature of relationships between individu- 
als or organizations. Sociograms — ^visual displays of 
network structure — are useful snapshots that convey this 
information. Figure 1 is a sociogram showing contact 
among leaders for one site in Mathematica’s Community 
Partnerships study described on page 2. 

Each node in Figure 1 represents a leader, and each line 
represents a reciprocal relationship between two leaders. 
(Actor A reports having contact with Actor B at least 
monthly and Actor B also reports having contact at least 
monthly with Actor A.) The absence of a tie between 
individuals indicates that at least one of them did not 
report any contact. 

Site A leaders had a distinct pattern for their relation- 
ships. Work group leaders (triangles) were largely 
unconnected in their contact with other leaders in the 
team. The core team members (squares) were con- 
nected with each other, while staff members (circles) 
were somewhat between the other two groups. If staff 
are supposed to be intermediaries between a strong 
decision-making body made up of core team mem- 
bers, this pattern suggests that the leadership team 





STUDYING LEADER'S STRUCTURE AND 
PROCESS HELPS SHED LIGHT ON OUTCOMES 



In Mathematica’s evaluation of the Community 
Partnerships for Older Adults program, the Robert 
Wood Johnson Foundation awarded four-year grants 
to eight communities to increase awareness of elder 
issues, improve the long-term care system, and 
advocate for policy change during the second round 
of this program. Partnerships in these communities 
include a broad array of organizations, providers, 
and consumers focused on aging issues and typically 
consist of representatives of public agencies, private 
service providers, nonprofit organizations, local busi- 
ness leaders, and individual older adults. 

We used SNA to study the leadership team as the net- 
work of interest, because it is responsible for identify- 
ing the partnership’s goals and carrying out activities 
to achieve them. We defined the team as including 
core team members, work group leaders, and project 
staff. The core team guides the partnership and sets 
its mission. Work groups engage in specific activi- 
ties, such as implementing projects. Project staff are 
funded by the grantee agency to coordinate activities. 

Our survey questions focused on two kinds of 
relationships: (1) communication between members, 
and (2) level of regard they had for each other. We 
asked leaders about frequency of contact with others, 
which leaders had good ideas, which leaders played 
important roles, and which leaders were productive. 



is functioning as intended. Similarly, if work group 
leaders are assigned specific tasks, such as setting 
up a hotline, and not expected to interact with other 
leadership team members. Figure 1 suggests they are 
functioning as planned. 

This pattern is not the same as that found in the study 
site illustrated in Figure 2. The contact among leaders 
at Site B suggests that staff members formed a hub 
around which the core team and work group leaders 
connected. One core team member (upper left) was 
isolated and had no reciprocal contacts with others. 

The qualitative analysis of sociograms can be 
augmented with descriptive statistics. The extent to 
which individuals interact with members of the same 
or different groups can be quantified and compared 
within and between partnerships for evaluation pur- 
poses (as we will illustrate). 

Identify prominent network members. Identifying 
members who are most involved in a network can 
be a useful part of process analysis. For example, an 
individual’s position in the network can influence 
how he or she sees it. Interviews among those who 
are more connected can provide different insights 



Figure 1: Contact at Least Monthly, Site A 




A= Work Group Leader 

into program operations, compared to interviews 
with individuals who are less connected. We can also 
confirm that power resides where we would expect it 
(if, for example, the chair of a core team is more con- 
nected to other influential individuals). 

In SNA, this concept of prominence is rooted in cen- 
trality — individuals or organizations are influential 
or prominent because they hold strategic positions 
or have many ties or groupings in a network. We can 
also identify the relative importance of members by 
their characteristics, such as the type of organization 
they represent, amount of experience they have, or 
the roles they play. 

Since the leadership teams in our study have spe- 
cific roles, the process analysis can explore which 
groups are more prominent. For example, do staff 
members play a brokerage role between leadership 
team members, or are they more central among all 
the members? Are core team members as prominent 
as staff? Not having any core team members who are 
prominent in a network may suggest a more central- 
ized approach in which the grantee holds power. 

Site A leaders with the most relationships in Figure 1 
(represented by nodes in the sociogram with the most 
lines or connections) were the most prominent in the 
network. Two staff members and several core team 
members were important in this sense. In contrast, 
work group leaders were not prominent — only one had 
more than one relationship. Moreover, the core team 
chair (not identified in the figure) had contact with few 
individuals and was on the periphery of the sociogram. 

Target interventions to improve the network. We 

can compare SNA results with a program’s design (or 
a logic model or theory of change) to assess how well 
the existing network matches the program intent. This 
comparison to an “ideal” network can suggest ways 
the network could be improved. 
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Figure 2: Contact at least Monthly, Site B 




Table 1 presents a different view of the leadership 
teams from Figures 1 and 2. Here, we included all 
data about leaders (including nonrespondents to the 
survey) and did not restrict relationships to those that 
were reciprocated. The table shows density — the pro- 
portion of relationships that exist within and among 
different types of leaders. Density is calculated as 
the number of ties among members divided by all 
possible ties. Initiators are individuals that report a 
relationship; receivers are targets (and include leaders 
that did not respond to the survey). The informa- 
tion can be compared with the “ideal” relationships 
among leadership team members. 

For Site A, 91 percent of relationships that could 
exist from staff members to other staff members did 
exist, 61 percent of relationships from staff mem- 
bers to core team members that could exist did, and 
39 percent of relationships from staff members to 
work group members that could exist did. This same 
pattern — more relationships with staff members, 
fewer relationships with work group members — ^was 
found for core team and work group leaders. 

The information in Table 1 confirms the visual 
relationships seen in Figures 1 and 2. Work group 
members in Site A were less connected to the leader 
network, as shown by the less frequent relationships 
reported by work group members to other leaders, 
and the less frequent relationships reported from 
other leaders to work group leaders. Compared to 
Site A, Site B staff had more frequent contact to and 
from leaders, as did work group members. Core team 
members had relatively less contact. 

Provide formative feedback. The results produced 
through social network analysis can be shared with sites 
through a formative feedback process. This process can 
confirm that the network picture reflects the views of 
people in the network (and if not, how the views differ). 
Visual representations of network operations can be 
compelling, both for evaluators and for program admin- 



TABLE 1 



AMOUNT OF CONTACT AT LEAST MONTHLY AMONG STAFF, 
CORE TEAM, AND WORK GROUP LEADERS 


1 Receivers 


Initiators 


Staff 


Core 

Team 


Group 


Site A 








Staff 


91% 


61% 


39% 


Core Team 


81% 


62% 


19% 


Work Group 


36% 


6% 


7% 


Site B 








Staff 


100% 


85% 


72% 


Core Team 


50% 


18% 


18% 


Work Group 


68% 


30% 


40% 



istrators. However, there are important ethical consider- 
ations in presenting results, depending on how the data 
were collected and the questions asked. ^ For example, 
results may need to be shown in aggregate or without 
identifiers to maintain anonymity. 

For each partnership in our study, we reviewed results 
with staff to verify whether these pictures reflected 
their views of the leadership. We also asked whether 
the visuals were accurate and what staff thought 
could be different. A fruitful area of discussion in- 
volved how work group leaders were integrated. Site 
A, in particular, had already identified lack of integra- 
tion as a problem and had been involving core team 
members more with work group activities. 

SNA in Outcomes Evaluation 

In addition to understanding processes, SNA can 
explore outcomes, either through examining relation- 
ships between networks and outcomes or observing 
how a network evolves as an outcome itself. 

Examine relationship between networks and 
outcomes. Networks observed through SNA pro- 
cess analysis can be compared with outcomes of 
interest to determine whether networks are related 
to outcomes. We could test, for example, whether 
leadership teams that are either more connected (that 
is, more dense) or that become more connected over 
time are more successful than teams that are less 
connected. We could also look at whether leadership 
teams with more integrated work group leaders are 
more likely to achieve their outcomes. 

In our study, the outcome measure in the first year of 
operation is perceived effectiveness — how effec- 

^ This issue is discussed further in Penuel, W.R., W. Sussex, C. 
Korbak, and C. Hoadley (2006), “Investigating the Potential Uses 
of Social Network Analysis in Educational Evaluation,” American 
Journal of Evaluation, 27(4), 437-451. 
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tive leaders feel they have been in working on their 
priorities. For the eight partnerships, members of 
the Site A leadership team had the lowest perceived 
effectiveness, while Site B had the highest. An 
important distinction between the two sites is how 
members regarded each other. Figure 3 illustrates 
how Site A leaders regarded each other (showing 
reciprocal relationships on a composite measure of 
who had good ideas, who was productive, and who 
played an important role). Figure 3 shows a similar 
pattern of relationships as those observed in Figure 
1 — core team members and work group leaders did 
not have high regard for each other. (This could 
reflect not being aware of each other or having nega- 
tive views.) For Site B, the pattern is quite different, 
both from the level of regard in Site A and from con- 
tact relationships in Site B (Figure 2). Though staff 
remained in the center of the relationships, leaders 
were connected with many other leaders — reflecting 
greater density. This pattern suggests that leaders 
had high regard for each other and their activities. 

Track network evolution. Developing visuals 
such as Figures 1 through 4 over time illustrates 
how a network evolves. The network itself could 
be the outcome of a program’s design — for in- 
stance, to build a coalition involving advocates in 
state health policy decisions or to encourage better 
relationships among child care agencies within a 
community. In these examples, we would observe 
whether, as the program ends, advocates or agen- 
cies are more or less likely to work with each other. 
We would also look at whether these relationships 
continue after initial funding ends. 

Part of Mathematica’s ongoing evaluation is tracking 
each partnership to examine how its leadership team 
changes over time, and whether staff prominence 
declines as partnerships become more integrated into 
their communities and initial funding ends. 

Making Connections 

SNA is a useful tool for evaluating programs in which 
relationships are important for outcomes. It allows 
evaluators to identify what the network is, how it 
operates, and how it affects program outcomes. In 
addition, SNA suggests ways for funders and partici- 
pants to make changes during implementation. 

Using SNA also presents challenges. Evaluators must 
understand the network they want to observe in order 
to define it and determine whether it is conducive to 
SNA methods. Survey nonreponse can be detrimen- 



Figure 3: Relationship Regard, Site A 




Figure 4: Relationship Regard, Site B 




A= Work Group Leader 

tal to analysis — even a small amount of missing data 
can greatly limit interpretation of results. Non-survey 
data — such as attendance records or other observa- 
tional data — can help address this problem, but the in- 
formation they provide may be limited. Small network 
samples can limit generalizability or interpretation 
of results. And maintaining confidentiality may need 
special attention, particularly when asking individuals 
to assess the quality of their relationships. 

Early on, proper planning and evaluation design can 
address these challenges. Social network methods 
provide compelling ways to investigate relational 
aspects of social structures and produce visual and 
mathematical analyses of complex human systems. 

For further information, contact researcher Todd Honeycutt at 
(609) 945-3397, thoneycutt@mathematica-mpr.com. 
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